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Afterschool programs are increasingly recognized as 
venues for effectively engaging children and youth in 
science, technology, engineering, and mathematics 
(STEM). Since the last set of national science standards was 
published in 1996, the number of afterschool programs 
and participants—and of dollars invested in STEM 
learning in these settings—has expanded substantially. 
The afterschool field has enthusiastically embraced STEM 
education. According to a 2011 Afterschool Alliance 
survey, a majority of providers now believe that it is 
important for them to offer STEM programming. 

As more stakeholders get involved in the effort to en¬ 
gage youth in STEM outside of school, afterschool provid¬ 
ers are being asked to document a wide range of outcomes, 
from generating interest in STEM to improving standard¬ 
ized test scores in math and science and to increasing the 


number of students who pursue STEM majors in college. 
Although stakeholders agree that afterschool STEM edu¬ 
cation can be powerful, there is less agreement on the 
critical question of which aspects of STEM education the 
afterschool field is best positioned to support (e.g., Sefton- 
Green, 2012). This issue has significant policy implications 
as lawmakers work on legislation that affects STEM edu¬ 
cation and decide on funding for efforts to improve it. 
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The Next Generation Science Standards (Achieve, 
2013) call for STEM education to move beyond facts and 
procedures by introducing broad concepts, such as scale 
or systems, and by engaging students in STEM practices 
such as developing evidence-based explanations. Be¬ 
cause this expanded perspective will require active and 
contextualized modes of learning, the afterschool field 
has a clear and compelling opportunity to position itself 
as a key partner in a STEM learning “ecosystem” com¬ 
prising schools, afterschool programs, and other com¬ 
munity settings and partners. 

In this context, greater clarity about appropriate af¬ 
terschool STEM learning goals and outcomes is essential 
to demonstrating how afterschool programs can best fa¬ 
cilitate STEM education. Elence, in spring 2012, the Af¬ 
terschool Alliance undertook a study to ask afterschool 
stakeholders what aspects of STEM learning the field is 
best positioned to support. The aim of the Afterschool 
STEM Outcomes Study was to identify consensus views 
on appropriate and feasible outcomes and indicators for 
afterschool STEM programs. The study provides a real¬ 
istic vision of the field’s potential for supporting student 
learning, a vision that can inform policy decisions and 
evaluation design. 

Listening to the Field 

The growth in afterschool STEM has been accompanied 
by expansion of the field of informal science education 
(ISE), where a significant body of work has accumu¬ 
lated to define youth outcomes and guide assessment. 
Most notably, Learning Science in Informal Environments, 
a report of the National Research 
Council (NRC, 2009), articulates 
the multi-faceted dimensions of 
science education, which involves 
not only scientific concepts and 
skills but also scientific practices, 
ways of knowing, fields of activ¬ 
ity, and the development of interest 
and identities. 

Other work has articulated 
ways to discern evidence of science 
learning (Friedman, 2008), for ex¬ 
ample, through changes in interest, 
skills, and patterns of behavior. In 
addition, work in the learning sci¬ 
ences has revealed how science literacy develops across 
settings and over time (Bransford et al., 2006; Ito et al., 
2012). Policy studies have posited the importance of in¬ 
tegrating the full array of institutional settings—schools, 


afterschool programs, and other cultural and community 
settings—to support STEM learning (Carnegie Corpora¬ 
tion of New York & Institute for Advanced Study, 2009; 
President’s Council of Advisors on Science and Technol¬ 
ogy, 2011). 

However, as many of these reports note, assessing the 
full impact of ISE projects is complex. Afterschool pro¬ 
grams face a particular challenge. They differ from other 
commonly discussed ISE settings, such as museums, 
mass media, and gaming environments, because they: 

• Sit at the junction of formal and informal learning settings 

• Are driven by strong youth development goals 

• Are commonly led, and their activities facilitated, by 
non-STEM experts 

• Are more likely than other non-school educational set¬ 
tings to work with young people from populations his¬ 
torically underrepresented in STEM fields (Afterschool 
Alliance, 2009) 

Not only are afterschool programs different from 
other ISE settings, they are also diverse among them¬ 
selves, varying in such particulars as: 

• Student ages, prior STEM experience, and interests 

• Staff members’ formal training and STEM background 

• Time dedicated to STEM, ranging from daily to a few 
times a year 

• Local resources and partnerships with, for example, 
universities, museums, or parks 

• Level of resources allocated to science programming 

• Type of STEM programming, from isolated hands-on 
activities to multi-year mentorships 

This diversity is an important 
asset. Key to a robust learning ecol¬ 
ogy is a wide array of opportunities 
for learners to develop and pursue 
new interests. However, the diver¬ 
sity complicates efforts to describe 
concisely the contributions of the 
field as a whole. As afterschool 
becomes more widely accepted as 
a partner in STEM education—and 
therefore subject to increased scru¬ 
tiny—the field must clearly articu¬ 
late how afterschool programs con¬ 
tribute to children’s STEM learning. 
The Afterschool STEM Outcomes Study, Defining 
Youth Outcomes for STEM Learning in Afterschool (After¬ 
school Alliance, 2013), used a process called the Delphi 
method. In this process, a carefully selected group of 
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experts answer questionnaires in multiple rounds. Af¬ 
ter each round, a facilitator summarizes responses; the 
experts are then encouraged to revise their answers in 
light of the replies of other panelists. During this process, 
the range of the responses is expected to decrease as the 
group converges toward consensus. The process ends at 
a pre-defined stop point: completing a certain number 
of rounds, achieving consensus, or getting stable results. 
Participants remain anonymous throughout the process. 

The Delphi method was chosen for the Afterschool 
STEM Outcomes Study to achieve convergence of opin¬ 
ions among two groups of experts: 

• A panel of 55 afterschool providers: experienced after¬ 
school leaders responsible for selecting, designing, or 
leading programming; providing professional develop¬ 
ment; and delivering program outcomes 

• A panel of 25 afterschool STEM supporters: funders; 
national education policy leaders; and state education 
department representatives responsible for providing 
funding, making policy decisions, and establishing 
outcomes for afterschool programs 

Three rounds of online surveys were conducted with 
each of these two groups in order to work toward con¬ 
sensus on: 

• The main outcomes for which the field as a whole 
could be responsible 

• The indicators of progress toward these outcomes 

• Specific sub-indicators that afterschool programs 
could document to demonstrate their contributions 
toward achieving these outcomes 

Afterschool STEM Outcomes Study Results 

The Afterschool STEM Outcomes Study yielded con¬ 
sensus about three major outcomes for children and 
youth in diverse afterschool STEM programs: developing 
young people’s interest in STEM, building their capac¬ 
ity to engage productively in STEM learning activities, 
and helping them come to value STEM. These broad 
developmental outcomes and indicators of learning reso¬ 
nate with prior literature on afterschool STEM programs 
(Afterschool Alliance, 2011), child and human develop¬ 
ment (e.g., Hidi & Renninger, 2006; Holland, Lachicotte, 
Skinner, & Cain, 1998; Lave & Wenger, 1991), youth 
development (e.g., Barber, Stone, Hunt, & Eccles, 2005; 
Eccles, 2005), and science learning (e.g., NRC, 2007, 
2009). For each of the three outcomes, participants iden¬ 
tified a set of indicators and sub-indicators, as shown in 
Table 1. Participants also indicated which outcomes they 
felt the field was best positioned to address. 


The results of the Afterschool STEM Outcomes 
Study provide a framework for thinking about how in¬ 
dividual programs can demonstrate their support for 
STEM learning. For example, many afterschool leaders 
believe that their programs support children’s ability to 
work in teams, but not all are aware that teamwork is 
intrinsic to STEM practices and professions. The After¬ 
school STEM Outcomes framework helps afterschool 
leaders connect STEM learning outcomes to the program 
goals they value. 

Although the expert panelists achieved consensus 
on the outcomes and indicators shown in Table 1, several 
interesting distinctions in their responses have implica¬ 
tions for policy and practice. 

STEM Activities vs. STEM Values 

In defining the outcomes that afterschool STEM pro¬ 
gramming is best positioned to affect, the expert consen¬ 
sus ranked the following three indicators highest: active 
participation in STEM learning opportunities; curiosity 
about STEM topics, concepts, or practices; and ability 
to productively engage in STEM processes of investiga¬ 
tion. Panelists also agreed that afterschool programs 
are less likely to be able to affect the other three indica¬ 
tors: awareness of STEM professions, ability to exercise 
STEM-relevant life and career skills, and understanding 
of the value of STEM in society. This finding suggests 
that the afterschool field is more confident about affect¬ 
ing indicators related to the active doing of STEM and 
less confident about affecting indicators that relate to the 
practices and value of STEM in society. 

Shorter-Term vs. Longer-Term Sub-Indicators 

In ranking sub-indicators, panelists indicated the most 
confidence that the field’s work supports young people’s 
interests, inquiries, and engagement with STEM activities. 
These sub-indicators of progress toward STEM learning 
can be seen and documented in immediate ways. The ex¬ 
perts felt comparatively less confident about achieving im¬ 
pacts described by some of the longer-term sub-indicators 
of learning such as demonstration of STEM knowledge, 
understanding of STEM methods of investigation, and pur¬ 
suit of further in-school or out-of-school STEM learning. 
The comparative lack of confidence about longer-term 
sub-indicators may reflect the uncertainty of partici¬ 
pant attendance and other inherent structural features 
of the afterschool setting. These structural features must 
be considered in policy decisions and evaluation design. 


44 Afterschool Matters 


Fall 2013 


Table 1. Framework for Developmental Outcomes and Learning Indicators for Afterschool STEM 


OUTCOME 
Through STEM 
afterschool programs, 
children and youth... 




A. 


INDICATOR 
You know or can see 
that children and 
youth demonstrate... 



Active participation 
in STEM learning 
opportunities 


SUB-INDICATORS 

If you had appropriate tools, you could 
document the following types of evidence... 


Active engagement and focus in STEM learning activities 

Examples of evidence: persisting in a task or program; sharing 
knowledge and ideas; expressing enthusiasm, joy, etc. 

Pursuit of other out-of-school time 
STEM learning opportunities 

Examples of evidence: enrolling in programs; attending programs 
regularly; reporting performing STEM-related activities at home 

Pursuit of school STEM learning opportunities 


Develop an interest 
in STEM and STEM 
learning activities 

"I like to do this." 


Examples of evidence: participating more actively in school 
STEM activities; enrolling in courses; selecting special programs 
or schools; improving academic achievement 


Curiosity about 
STEM topics, concepts, 
or practices 


Active inquiries into STEM topics, concepts, or practices 

Examples of evidence: exploring ideas verbally or physically; 
questioning, hypothesizing, testing 

Active information-seeking about mechanical or natural 
phenomena or objects 

Examples of evidence: conducting Internet searches for more 
information; getting books or journals about STEM; watching TV 
programs on science 



Ability to productively 
engage in STEM 
processes of 
investigation 


Demonstration of STEM knowledge 

Examples of evidence: demonstrating increase in knowledge in 
specific content areas; making connections with everyday world; using 
scientific terminology 
Demonstration of STEM skills 
Examples of evidence: formulating questions; testing, exploring, 
predicting, observing, collecting and analyzing data 
Demonstration of an understanding 
of STEM methods of investigation 
Examples of evidence: demonstrating understanding of the nature 
of science; using evidence-based reasoning and argumentation; 
demonstrating engineering design practices 


Develop a capacity to 
productively engage 
in STEM learning 
activities 

"I can do this." 


Demonstration of mastery of technologies 
and tools that can assist in STEM investigations 

Examples of evidence: developing capacity to use measurement and 
other scientific instruments; running computer programs for data 


Ability to exercise 
STEM-relevant life and 
career skills 


analysis; developing effective methods to communicate findings 
Demonstration of ability to work in teams 
to conduct STEM investigations 

Examples of evidence: communicating effectively with team members; 
collaborating effectively with team members; demonstrating 
leadership on the team 

Demonstration of applied problem-solving abilities 
to conduct STEM investigations 

Examples of evidence: engaging in critical thinking; questioning, 
sequencing, reasoning 
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OUTCOME 

Through STEM 
afterschool programs, 
children and youth... 

INDICATOR 

You know or can see 
that children and 
youth demonstrate... 

SUB-INDICATORS 

If you had appropriate tools, you could 
document the following types of evidence... 

Come to value the 
goals of STEM and 

STEM learning activities 

"This is important 
to me." 

Understanding 
of the value of STEM 
in society 

Demonstration of an understanding of the relevance 
of STEM to everyday life, including personal life 

Examples of evidence: referencing examples of STEM in everyday life, 
everyday problems 

Demonstration of knowledge of important civic, global, 
and local problems that can be addressed by STEM 

Examples of evidence: contributing to projects that address a 
community need; developing awareness of how STEM is implicated 
in larger societal issues 

Demonstration of awareness of opportunities 
to contribute to society through STEM 

Examples of evidence: engaging in a service-learning project 

Awareness of 

STEM professions 

Development of an understanding of the variety 
of STEM careers related to different fields of study 

Examples of evidence: gaining knowledge about relevant professions; 
gaining knowledge of where such jobs and careers exist 
Demonstration of knowledge of how to pursue STEM careers 
Examples of evidence: acquiring knowledge of what courses are 
needed to prepare for or pursue STEM degrees; declaring STEM 
interests or majors 

Demonstration of awareness that STEM is accessible to all 

Examples of evidence: expressing a desire to meet role models; 
declaring STEM interests and majors; desiring to become 
a role model to pave the way for others 


Availability of Assessment Tools 

When asked about the availability of assessment tools to 
document the learning outcomes and indicators, the af¬ 
terschool STEM supporters—state and national education 
leaders and funders—were much more optimistic about 
the availability of such tools than were the afterschool pro¬ 
viders. This difference, though it did not meet the crite¬ 
rion for statistical significance, suggests that the two expert 
groups may have different standards for assessment. An¬ 
other possibility is that providers are unaware of existing 
tools or feel that these tools are not usable or not accessible. 

In-School vs. Out-of-School STEM Learning 

Panelists were least confident that the afterschool field 
could demonstrate effects regarding the sub-indicator “pur¬ 
suit of school STEM learning opportunities.” This result is 
extremely important in light of the fact that many large- 
scale studies have used school achievement measures to as¬ 
sess the contributions of afterschool programs to children’s 


learning. The relatively low ranking of this sub-indicator 
may reflect the panelists’ feeling that achievement test 
scores are affected by too many factors that are out of 
the control of afterschool practitioners and supporters. 

Implications and Areas for Further Research 

The outcomes, indicators, and sub-indicators iden¬ 
tified by participants in the Afterschool STEM Out¬ 
comes Study as representative of the field’s contribu¬ 
tions to STEM learning constitute a good step toward 
articulating the impact of afterschool STEM programs. 
The field’s continuing challenge is to develop tools and 
methods that can document outcomes without sig¬ 
nificantly interfering with the afterschool experience, 
as, for example, pen and paper tests might do, and 
without incurring significant cost, as, for example, 
conducting ethnographic research might do. Besides 
documenting outcomes, the field is also challenged 
to show how program activities contribute to those 
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RELATING THE AFTERSCHOOL STEM OUTCOMES FRAMEWORK 
TO A MIDDLE SCHOOL STEM PROGRAM 


The example of Exploratorium XTech helps to demonstrate how the Afterschool STEM 
Outcomes framework relates youth leadership, a capacity highly valued in many afterschool 
and youth programs, to STEM learning. 

Exploratorium XTech is a three-year program that works with middle school youth during 
summer camps and on Saturdays during the school year. XTech students design and construct 
table-top versions of exhibits or activities on the museum floor. In the process, they engage 
with phenomena; develop proficiencies with machine and digital tools; and exercise scientific 
practices including design, experimentation, problem-solving, observing, and analyzing. 

As they grow older, XTech students can become XTech facilitators, assisting staff in XTech and 
in community afterschool programs to lead design-and-build activities with younger students. 
Exercising leadership in XTech involves developing mastery of STEM concepts, tools, and 
practices in order first to support incoming students and eventually to take on formal teaching 
roles with elementary-aged students. In the process, students develop their interests and 
become local experts on particular tools, such as band saws or video documentation, or on 
concepts, such as mechanics or optics. Led by their interests, they deepen their engagement 
and mastery, as evidenced by the increasing complexity of the mini-exhibits they produce over 
time. Developing participants' capacities to engage in STEM is not the end goal but the means 
for full program participation and leadership development. 

In a focus group, a number of participants shared their reflections on how it felt to teach 
younger students. One commented: 

I kind of think the whole thing was fun because, in the beginning, we were . . . trying to 
tell ourselves, "Okay, this is what we can teach," and we would teach. What always came 
up is, "Okay, but we have to explain this to kids who are way younger than us." That's 
what made it more interesting, was taking all of these concepts we knew and turning it 
into a way to explain it to third through fifth graders. (Vossoughi, 2012, p. 8) 

Although youth leadership is not included in the Afterschool STEM Outcomes framework, 
the framework clarifies the ways in which STEM activities provide a context in which 
leadership can develop. 


OUTCOME 

INDICATOR 

SUB-INDICATORS 

XTECH ACTIVITIES 

Developing interest 

Active participation 
in STEM learning 
opportunities 

Active engagement 
and focus in STEM 
learning activities 

Mastering ideas and 
techniques in order 
to teach others 

Developing capacities 

Ability to 

productively engage 
in STEM processes of 
investigation 

Demonstration of 

STEM knowledge; 
demonstration of 

STEM skills 

Designing and 
constructing 
increasingly 
sophisticated STEM 
mini-exhibits 


Krishnamurthi, Bevan, Rinehart, & Coulon 


WHAT AFTERSCHOOL STEM DOES BEST 47 













outcomes. The nature of children’s experience in after¬ 
school programs remains largely unexamined. 

The expert panelists agreed that afterschool programs 
contribute to students’ school success, but they expressed 
lower levels of confidence, relative to other effects, in the 
field’s ability to affect school outcomes such as test scores 
and course taking. This finding is perhaps unsurprising, 
given the peripheral relationship of school outcomes to 
afterschool interventions, but it is notable because pro¬ 
grams have often been evaluated on exactly (and some¬ 
times only) these dimensions. Better communication and 
coordination between school and afterschool, together 
with a clearer understanding of what each can contribute 
as a component in an ecosystem of learning, would help 
to correlate learning between the two. 

The study results point to some key areas that could 
benefit from additional research. For one, the panelists’ 
relative confidence in the field’s ability to document im¬ 
mediate as opposed to longer-term effects, along with 
their lack of confidence about demonstrating impact on 
school STEM learning, suggest the need for new research 
and evaluation methodologies and instruments. The field 
needs ways to investigate STEM learning across settings, 
showing how immediate STEM learning outcomes in af¬ 
terschool settings relate to longer-term learning in school 
or other community settings. New tools may enable the 
field to articulate and evaluate the value and contribu¬ 
tions of afterschool programs. 

Another area for additional research is suggested 
by disparities between how panelists ranked particular 
learning indicators as opposed to the outcomes the in¬ 
dicators support—for example, indicating that programs 
do support teamwork but do not support the develop¬ 
ment of STEM skills. This finding suggests a need for 
dialogue and professional development that unpack core 
ideas in STEM learning outcomes. This need is especially 
relevant and important in relation to the concepts and 
cross-cutting practices highlighted in the Next Genera¬ 
tion Science Standards, which will require an integrated 
view of teaching and learning. Dialogue about learning 
outcomes can help afterschool providers understand the 
many dispositions and practices involved in STEM fields 
and how programs can support their development. 

Finally, variations in perspective between afterschool 
supporters and afterschool providers—though these did 
not reach the level of statistical significance—invite further 
investigation to detect and resolve any real and meaningful 
differences that emerge between the two groups. For ex¬ 
ample, supporters felt more confident than providers that 
programs could support development of students’ STEM 


knowledge. What is the basis of this difference, and how 
does the disparity in perspectives affect how programs are 
designed and evaluated? Following up on this issue should 
yield information to help the field move forward to achieve 
its full potential, ft may also provide guidance to funders 
seeking areas for high-impact afterschool investments. 

The Place of Afterschool 
in the STEM Ecosystem 

An increasingly robust research base points to the need to 
build an ecosystem of learning that spans school and out- 
of-school learning (Bevan & Michalchik, 2013; Ito et al., 
2013). Concurrently, the ability of the afterschool field to 
support STEM learning is advancing at a rapid pace. This 
convergence brings a unique opportunity to cement the 
role of afterschool programming as an integral component 
of a re-imagined effort to improve STEM education. 

A clear understanding of the outcomes to which after¬ 
school STEM programs can contribute is essential to mak¬ 
ing this case. The consensus produced by the Afterschool 
STEM Outcomes Study provides on-the-ground perspec¬ 
tives—from those who lead, design, and fund afterschool 
STEM programs—about the outcomes the field is best 
positioned to advance. Policymakers and funders should 
consider these consensus views in framing the place of 
afterschool as an important part of the STEM education 
ecosystem. Furthermore, practitioners and researchers 
can use these outcomes and their corresponding indica¬ 
tors to design evaluations that document the role of after¬ 
school programs in the STEM learning ecology. 

The diversity of afterschool STEM programs is simul¬ 
taneously a strength and an argument against developing 
one unifying measure for use across settings. Eiowever, 
using a common language like that provided by the After¬ 
school STEM Outcomes Study could facilitate a synthesis 
of results from like programs while enabling description 
of the range of possible outcomes across diverse pro¬ 
grams. This common framework can enable the field to 
better describe how afterschool programs help children 
develop interest in, build capacity for, and come to value 
STEM and STEM learning activities. 
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